Systems and methods for electronic messaging testing optimization in prospect electronic messages series

ABSTRACT

Disclosed are methods, systems, and non-transitory computer-readable medium for optimizing user experience with respect to email journeys. For example, a method may include determining a first development subset, a first quality assurance subset, and a first production subset of a first plurality of user identifiers; determining and transmitting a first development set of messages, a first quality assurance set of messages, and a first production set of message; determining a development performance indicator, a quality assurance performance indicator, and a production performance indicator based on one or more messages received in response to the transmitted first development set of messages, first quality assurance set of messages, and first production set of message, respectively.

TECHNICAL FIELD

Various embodiments of the present disclosure generally relate tooptimizing user experience, and more particularly, to optimizing userexperience with respect to email journeys.

BACKGROUND

Conventional methods of testing and optimizing email journeys fail tomitigate risk for poor performing tests in a nimble and agile manner.Additionally, some conventional methods are onerous to setup whichreduces testing throughput and introduces additional risk from humanerror.

A first example of a conventional method includes setting up amulti-pathed email journey within a marketing platform (e.g., asalesforce marketing cloud). While this method may enable the creationof multiple paths within an email journey and facilitate testing, thismethod is difficult to setup, which thereby increases the likelihood ofhuman error and reduces testing throughput. Additionally, this method ismonolithic and inflexible due to the fact that all variations of controlhave to be setup at a same time and the sequence also has to be startedat the same time.

A second example of a conventional method includes setting up multipleatomic email journeys. While this method may reduce the setup complexityand eliminate concurrent start requirements, it introduces a risk thatentrants to the email journeys may be duplicated across email journeys.Accordingly, this may result in entrants receiving multiple emailjourney emails.

Accordingly, there remains a significant need for email journeyoptimization to maximize testing throughput while minimizing the risk ofa poorly performing test negatively impacting overall results inaddition to minimizing the risk of human setup error and negative userexperience outcomes such as receiving duplicated emails.

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart, or suggestions of the prior art, by inclusion in this section.

SUMMARY OF THE DISCLOSURE

According to certain aspects of the disclosure, systems and methods aredisclosed for optimizing user experience to overcome the problems withconventional methods noted above.

In one embodiment, a computer-implemented method comprises: determining,by one or more processors, a first development subset, a first qualityassurance subset, and a first production subset of a first plurality ofuser identifiers, wherein each of the first plurality of useridentifiers is associated with a customer; determining, by the one ormore processors, a first development set of messages; transmitting, bythe one or more processors, the first development set of messages to thefirst development subset of user identifiers; receiving, by the one ormore processors, one or more messages in response to the firstdevelopment set of messages transmitted to the first development subsetof user identifiers; determining, by the one or more processors, adevelopment performance indicator based on the one or more messagesreceived in response to the first development set of messagestransmitted to the first development subset of user identifiers;determining, by the one or more processors, a first quality assuranceset of messages; transmitting, by the one or more processors, the firstquality assurance set of messages to the first quality assurance subsetof user identifiers; receiving, by the one or more processors, one ormore messages in response to the first quality assurance set of messagestransmitted to the first quality assurance subset of user identifiers;determining, by the one or more processors, a quality assuranceperformance indicator based on the one or more messages received inresponse to the first quality assurance set of messages transmitted tothe first quality assurance subset of user identifiers; determining, bythe one or more processors, a first production set of messages;transmitting, by the one or more processors, the first production set ofmessages to the first production subset of user identifiers; receiving,by the one or more processors, one or more messages in response to thefirst production set of messages transmitted to the first productionsubset of user identifiers; and determining, by the one or moreprocessors, a production performance indicator based on the one or moremessages received in response to the first production set of messagestransmitted to the first production subset of user identifiers.

In accordance with another embodiments, a computer system comprises: adata storage device storing processor-readable instructions; and aprocessor configured to execute the instructions to perform a method.The method may include: determining a first development subset, a firstquality assurance subset, and a first production subset of a firstplurality of user identifiers, wherein each of the first plurality ofuser identifiers is associated with a customer; determining a firstdevelopment set of messages; transmitting the first development set ofmessages to the first development subset of user identifiers; receivingone or more messages in response to the first development set ofmessages transmitted to the first development subset of useridentifiers; determining a development performance indicator based onthe one or more messages received in response to the first developmentset of messages transmitted to the first development subset of useridentifiers; determining a first quality assurance set of messages;transmitting the first quality assurance set of messages to the firstquality assurance subset of user identifiers; receiving one or moremessages in response to the first quality assurance set of messagestransmitted to the first quality assurance subset of user identifiers;determining a quality assurance performance indicator based on the oneor more messages received in response to the first quality assurance setof messages transmitted to the first quality assurance subset of useridentifiers; determining a first production set of messages;transmitting the first production set of messages to the firstproduction subset of user identifiers; receiving one or more messages inresponse to the first production set of messages transmitted to thefirst production subset of user identifiers; and determining aproduction performance indicator based on the one or more messagesreceived in response to the first production set of messages transmittedto the first production subset of user identifiers.

In accordance with another embodiment, a non-transitorycomputer-readable medium containing instructions that, when executed bya processor, may cause the processor to perform a method. The method mayinclude determining a first development subset, a first qualityassurance subset, and a first production subset of a first plurality ofuser identifiers, wherein each of the first plurality of useridentifiers is associated with a customer; determining a firstdevelopment set of messages; transmitting the first development set ofmessages to the first development subset of user identifiers; receivingone or more messages in response to the first development set ofmessages transmitted to the first development subset of useridentifiers; determining a development performance indicator based onthe one or more messages received in response to the first developmentset of messages transmitted to the first development subset of useridentifiers; determining a first quality assurance set of messages;transmitting the first quality assurance set of messages to the firstquality assurance subset of user identifiers; receiving one or moremessages in response to the first quality assurance set of messagestransmitted to the first quality assurance subset of user identifiers;determining a quality assurance performance indicator based on the oneor more messages received in response to the first quality assurance setof messages transmitted to the first quality assurance subset of useridentifiers; determining a first production set of messages;transmitting the first production set of messages to the firstproduction subset of user identifiers; receiving one or more messages inresponse to the first production set of messages transmitted to thefirst production subset of user identifiers; and determining aproduction performance indicator based on the one or more messagesreceived in response to the first production set of messages transmittedto the first production subset of user identifiers.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1 depicts an example environment in which methods, systems, andother aspects of the present disclosure may be implemented.

FIG. 2 depicts a schematic diagram depicting an architecture of anexemplary environment for a clothing-as-a-service electronic platform,according to one or more embodiments.

FIG. 3 depicts an exemplary method of optimizing user experience,according to exemplary embodiments of the present disclosure.

FIGS. 4A-4B depict exemplary methods of utilizing one or more differentgroups of cohorts according to exemplary embodiments of the presentdisclosure.

FIG. 5 depicts an exemplary method of optimizing user experience,according to exemplary embodiments of the present disclosure.

FIG. 6 depicts an exemplary method of optimizing user experience,according to exemplary embodiments of the present disclosure.

FIG. 7 depicts an exemplary computer device or system, in whichembodiments of the present disclosure, or portions thereof, may beimplemented.

DETAILED DESCRIPTION OF EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thedisclosure, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

The present disclosure enables a configuration of multiple concurrenttests including one or more email journeys, e.g., an email journey maybe referred to as a sequence of emails evaluated both individually andcollectively as a group, while utilizing dynamic traffic allocation tomodify a percentage of incoming daily prospect cohort to each of theconcurrent tests based on test results accrued prior to receiving thecurrent prospect cohort.

The present disclosure provides a significant advantage in maximizingtesting throughput, which increases the speed of learning andoptimization, while minimizing risk, e.g., over-all business risk, froma poorly performing email journey. Additionally, the present disclosureprovides another significant advantage in reducing risk due to humanerror while enabling a high testing throughput rate. Furthermore, thepresent disclosure provides a significant advantage in enabling granularcontrol of a prospect cohort exposed to a test variant.

In some embodiments, the present disclosure may utilize separateenvironments to develop new features and/or concepts, test a discreteset of changes (for example, discrete changes to email journeys), andrelease the discrete set of changes to the public. In some embodiments,the present disclosure may determine one or more production emailjourney sets, quality assurance email journey sets, and/or developmentemail journey sets, as will be described in further detail below. Theproduction email journey set may comprise an optimized set of emails.The quality assurance email journey set may comprise the next candidatesfor the optimized set of emails pending validation. The developmentemail journey sets may comprise another set of emails for testing.

In some embodiments, the present disclosure may determine one or moregroups based on new prospects from a previous day. For example, the newprospects may be divided into three groups: (1) a development set, (2) aquality assurance set, and (3) a production set. The production setemail journey may be allocated to the subset of the prospects assignedto the production set, the quality assurance set email journey may beallocated to the subset of the prospects assigned to the qualityassurance set, and the development set email journey may be assigned tothe subset of the prospects assigned to the development set.Accordingly, the prospects assigned to the production set may experiencethe production set email journey, the prospects assigned to the qualityassurance set may experience the quality assurance set email journey,and the prospects assigned to the development set may experience thedevelopment set email journey.

The significant advantages provided by the present disclosure include:(1) dynamic traffic allocation to the concurrent test flows may beadjusted on a daily basis for the following day's incoming cohort basedon a performance of each of the tests; (2) each of the test environmentsmay be derived from a replica of the production set email journeyallowing for compartmentalized change and risk mitigation; (3) avoidsmonolithic test structures which reduces complexity and risk of humanerror; (4) risk mitigation through traffic allocation to ensure thatprospect cohorts are mutually exclusive and exhaustive; and (5)progressive allocation of traffic with higher percentages of the totaleligible cohort inversely proportional to the risk encapsulated by thatenvironment. For example, the higher risk development environments mayhave lower over-all traffic to validate new concepts, as will bedescribed in further detail below.

While the exemplary system architecture as described in the presentdisclosure relates to electronic transactions for subscribing to,purchasing, or renting wearable items (e.g., clothing-as-a-service(CaaS) or Try-Then-Buy (TTB) service), implementations disclosed hereinmay effectively serve various other online transactions in the contextof any other subscription, purchase, rental, or retail services withoutdeparting from the scope of the disclosure, such as, for example,subscribing to or making purchases in a software service, cleaningservice, delivery service, maintenance service, rental product, rentalvehicles, etc. In addition, while some descriptions and examplesdisclosed in the present disclosure refer to certain exemplarytransactions as transactions pertaining to “apparel” or “garments,” allof those transactions may effectively serve any wearable item (e.g., anarticle of clothing, apparel, jewelry, hat, accessories, or any otherproduct which may be worn), or even hospitality linens, consumer goods,or any other textile fabrics, without departing from the scope of thedisclosure.

As used in the present disclosure, the term “CaaS” (i.e.,clothing-as-a-service) may collectively refer to computer-implementedservices and functions associated with subscription, purchase, and/orrental services for users (e.g., periodic subscription for receivingwearable items, apparel rental or purchase order, distribution, returnprocessing, TTB services, account management, marketing, customerservice, warehouse operations, etc.). As used in the present disclosure,the term “wearable item” may refer to any article of clothing, apparel,jewelry, hat, accessories, or other product which may be worn by aperson, an animal, or a thing, or be used as an ornament for a person,an animal, or a thing. As used herein, the term “closeting” or “tocloset” may refer to a computer-implemented operation of placing one ormore garments into a virtual closet (e.g., a cart, a repository, or anytype of space which may be virtually associated with a particular set ofone or more garments for a future transaction). Additionally, “matching”may refer to a computer-implemented operation of determining a set ofone or more garments for allocating to a user and/or determiningwearability metrics for given garments, and “allocating” or “allocation”may refer to a computer-implemented operation of determining thegarments that should be assigned and shipped to one or more particularusers.

In accordance with the present disclosure, user interfaces, periodicallyexecuted computer-implemented services, ad hoc services, and automationsbeing integrated together in a connected platform may be achieved by auniquely configured system architecture, job execution clusterconfiguring one or more processors to perform both storefront and backoffice tasks, and various user interfaces providing specialized orcustomized access to users of different roles. For example, the systemmay periodically collect vast amounts of data attributes from historicaltransactions, form data sets indicative of each user's relationship withcertain apparel (e.g., a binary flag of whether a shipped garment wasactually worn by a user) in the back end, and train a neural networkwith those data sets to make specific front-end user recommendationswith highly wearable apparel. The ordered combination of various ad hocand automated tasks in the presently disclosed platform necessarilyachieve technological improvements through the specific processesdescribed more in detail below. In addition, the unconventional andunique aspects of these specific automation processes represent a sharpcontrast to merely providing a well-known or routine environment forperforming a manual or mental task.

The subject matter of the present description will now be described morefully hereinafter with reference to the accompanying drawings, whichform a part thereof, and which show, by way of illustration, specificexemplary embodiments. An embodiment or implementation described hereinas “exemplary” is not to be construed as preferred or advantageous, forexample, over other embodiments or implementations; rather, it isintended to reflect or indicate that the embodiment(s) is/are “example”embodiment(s). Subject matter can be embodied in a variety of differentforms and, therefore, covered or claimed subject matter is intended tobe construed as not being limited to any exemplary embodiments set forthherein; exemplary embodiments are provided merely to be illustrative.Likewise, a reasonably broad scope for claimed or covered subject matteris intended. Among other things, for example, subject matter may beembodied as methods, devices, components, or systems. Accordingly,embodiments may, for example, take the form of hardware, software,firmware, or any combination thereof (other than software per se). Thefollowing detailed description is, therefore, not intended to be takenin a limiting sense.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in one embodiment” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterinclude combinations of exemplary embodiments in whole or in part.

The terminology used below may be interpreted in its broadest reasonablemanner, even though it is being used in conjunction with a detaileddescription of certain specific examples of the present disclosure.Indeed, certain terms may even be emphasized below; however, anyterminology intended to be interpreted in any restricted manner will beovertly and specifically defined as such in this Detailed Descriptionsection. Both the foregoing general description and the followingdetailed description are exemplary and explanatory only and are notrestrictive of the features, as claimed.

In this disclosure, the term “based on” means “based at least in parton.” The singular forms “a,” “an,” and “the” include plural referentsunless the context dictates otherwise. The term “exemplary” is used inthe sense of “example” rather than “ideal.” The term “or” is meant to beinclusive and means either, any, several, or all of the listed items.The terms “comprises,” “comprising,” “includes,” “including,” or othervariations thereof, are intended to cover a non-exclusive inclusion suchthat a process, method, or product that comprises a list of elementsdoes not necessarily include only those elements, but may include otherelements not expressly listed or inherent to such a process, method,article, or apparatus. Relative terms, such as, “substantially” and“generally,” are used to indicate a possible variation of ±10% of astated or understood value.

Referring now to the appended drawings, FIG. 1 shows an exampleenvironment 100, according to one or more embodiments of the presentdisclosure. As shown, the example environment 100 may include one ormore networks 101 that interconnect a server system 102, user devices112, employee devices 116, tenant devices 120, and external systems 122.The one or more networks 101 may be, for example, one or more of acellular network, a public land mobile network, a local area network, awide area network, a metropolitan area network, a telephone network, aprivate network, an ad hoc network, an intranet, the Internet, a fiberoptic based network, a cloud computing network, etc. User devices 112may be accessed by users 108, employee devices 116 may be accessed byauthorized employees 114, and tenant devices 120 may be accessed byemployees of tenant entities 118. In some implementations, employeedevices 116 may be used to perform the functions of the tenant devices120 and/or the user devices 112. Server system 102 may comprise one ormore servers 104 and one or more databases 106, which may be configuredto store and/or process a plurality of data, microservices, and servicecomponents, and/or associated functions thereof, as described in moredetail below with respect to FIG. 2.

Users 108 may access the server system 102 through the one or morenetworks 101 using user devices 112. Each device among the user devices112 may be any type of computing device (e.g., personal computingdevice, mobile computing devices, etc.) which allows users 108 todisplay a web browser or a web-based application for accessing theserver system 102 through the network 101. The user devices 112 may, forexample, be configured to display a web browser, a web-basedapplication, or any other user interface (e.g., one or more mobileapplications) for allowing users 108 to exchange information with otherdevice(s) or system(s) in the environment 100 over the one or morenetworks 101. For example, a device among the user devices 110 may loadan application with a graphical user interface (GUI), and theapplication may display on the GUI one or more apparel recommendationsfor closeting by the user. Users 108 accessing user devices 112 may be,for example, users and/or potential users of apparel made available forsubscription-based distribution via electronic transactions and physicalshipment. Additionally, or alternatively, users 108 may access userdevices 112 to, for example, manage one or more user accounts, viewcatalogs, configure one or more user profiles, engage in customerservice communications, make purchase orders, track shipments, generateshipments, monitor order fulfillment processes, initiate or processreturns, order apparel for purchase, provide feedback, refer otherusers, navigate through various features such as size advisor, performpersonalized discovery, and/or make recommendations.

Employee devices 116 may be configured to be accessed by one or moreemployees 114, including, for example, customer service employees,marketer employees, warehouse employees, analytics employees, or anyother employees who are authorized and/or authenticated to performtasks, operations, and/or transactions associated with the server system102, and/or the external systems 122. In one embodiment, employeedevices 116 are owned and operated by the same entity or at least anaffiliate of the entity operating the e-commerce (e.g., CaaS) businesshosted on server systems 102. Each device among the employee devices 116may be any type of computing device (e.g., personal computing device,mobile computing devices, etc.). The employee devices 116 may allowemployees 114 to display a web browser or an application for accessingthe server system 102 and/or the external systems 122, through the oneor more networks 101. For example, a device among the one or more of theemployee devices 116 may load an application with graphical userinterface (GUI), and the application may display on the GUI one or morewarehouse operations associated with providing CaaS to users 108. Insome implementations, the employee devices 116 may communicate directlywith the server system 102 via communications link 117 bypassing publicnetworks 101. Additionally, or alternatively, the employee devices 116may communicate with the server system 102 via network 101 (e.g., accessby web browsers or web-based applications).

Tenant devices 120 may be configured to be accessed by one or moretenants 118. Each device among the tenant devices 120 may be any type ofcomputing device (e.g., personal computing device, mobile computingdevices, etc.). As used herein, each tenant, among one or more tenants118, may refer to an entity that allocates and/or supplies one or morespecific collections of apparel for the CaaS inventory. For example,each of the one or more tenants 118 may be a retailer, a designer, amanufacturer, a merchandizer, or a brand owner entity that supplies oneor more collections of wearable items to the CaaS inventory managedand/or accessed by the server system 102. In some embodiments, tenants118 may use one or more electronic tenant interfaces (e.g., a catalogcontent management system associated with each tenant) to provide theserver system 102 with wearable item data that describe apparel orwearable items made available for electronic transactions on serversystem 102. For example, one or more catalogs for each of the one ormore tenants 118 may be generated and/or updated at the server system102 dynamically and/or periodically. Tenant devices 120 may serve asaccess terminals for the tenants 118, for communicating with theelectronic tenant interfaces and/or other subsystems hosted at theserver system 102. The tenant devices 120 may, for example, beconfigured to display a web browser, an application, or any other userinterface for allowing tenants 118 to load the electronic tenantinterfaces and/or exchange data with other device(s) or system(s) in theenvironment 100 over the one or more networks 101.

External systems 122 may be, for example, one or more third party and/orauxiliary systems that integrate and/or communicate with the serversystem 102 in performing various CaaS tasks. Specific examples of theexternal systems 122 are described in detail below with respect to FIG.2. External systems 122 may be in communication with other device(s) orsystem(s) in the environment 100 over the one or more networks 101. Forexample, external systems 122 may communicate with the server system 102via API (application programming interface) access over the one or morenetworks 101, and also communicate with the employee devices 116 via webbrowser access over the one or more networks 101.

As indicated above, FIG. 1 is provided merely as an example. Otherexamples that differ from the example environment 100 of FIG. 1 arecontemplated within the scope of the present embodiments. In addition,the number and arrangement of devices and networks shown in system 100are provided as an example. In practice, there may be additionaldevices, fewer devices and/or networks, different devices and/ornetworks, or differently arranged devices and/or networks than thoseshown in system 100. Furthermore, two or more devices shown in FIG. 1may be implemented within a single device, or a single device shown inFIG. 1 may be implemented as multiple, distributed devices.Additionally, or alternatively, one or more devices may perform one ormore functions of other devices in the example environment 100. Forexample, employee devices 116 may be configured to perform one or morefunctions of tenant devices 120, in addition to their own functions.

FIG. 2 depicts a schematic diagram of an exemplary architecture 200 fora clothing-as-a-service electronic platform, according to one or moreembodiments. The components of the architecture may be accessed byauthorized terminals, such as employee devices 202 and user devices 204,over the one or more networks 101 or via any one or more other types ofnetwork (e.g., a cellular network, a public land mobile network, a localarea network, a wide area network, a metropolitan area network, atelephone network, a private network, an ad hoc network, an intranet,the internet, a fiber optic based network, a cloud computing network,etc.). As used herein, the user devices 204 may correspond to any one ormore user devices 112 depicted in FIG. 1, and the employee devices 202may correspond to any one or more of the employee devices 116 and/ortenant devices 120 depicted in FIG. 1.

As shown in FIG. 2, in general, architecture 200 may comprise aninternal system 206, external consumer facing apps 208, a webapplication hosting server 210, external systems 212, and tenant dataand analytics systems 214. Internal system 206 may comprise internalapps for operations, API endpoints, app endpoints, login endpoints,tenant aware services, tenant aware transactional data stores, backoffice jobs, data warehouse systems, file stores-snapshots, and 3rdparty data ETL processes.

In operation, employee devices 202 may access the internal system 206,which may be stored at networked, distributed, and/or local systems(e.g., one or more virtual private clouds and/or one or more physicalprivate networks). As used herein, a virtual private cloud may refer toa configurable pool of shared computing resources within a cloudenvironment, with groups of shared computing resources being allocatedfor a particular job(s), user(s) and/or a purpose(s). The internalsystem 206 may represent a virtual private cloud allocated for hostingthe entirety of the internal system 206. Additionally, the internalsystem 206 may include multiple virtual private clouds (e.g., subsets ofthe larger virtual private cloud encompassing the internal system 206),each allocated for one or more particular tasks, users, or purposes(e.g., tenant aware services, back office jobs, and data warehousesystems).

The employee devices 202 may communicate with the internal system 206through one or more internal apps 216 comprising one or more web-basedportals or user interfaces for operations. The one or more internal apps216 may be, for example, an electronic warehouse operations portaland/or an electronic administrative portal. The one or more internalapps 216 may include one or more web-based user interfaces, such as, forexample, one or more mobile applications compatible with predeterminedmobile device operating systems, a software application developed fordesktop operating systems, and/or a web site configured for browseraccess to web pages via one or more networks (e.g., the Internet or anintranet). Employees 116 or tenants 118 may use employee devices 202 toaccess the internal system 206 and perform various CaaS functionsrelated to warehouse operations and/or administrative tasks.

The internal system 206 may also include Application ProgrammingInterface (“API”) endpoints, APP (application) endpoints, and loginendpoints. In some implementations, these endpoints may be hosted in avirtual private cloud or a physical private network with a preconfiguredlevel of isolation from other virtual private clouds or physical privatenetworks hosting different components of the internal system 206. TheAPI endpoints may be locations from which the APIs can access resourcesof the internal system 206 that the APIs need to carry out theirfunctions. For example, the API endpoints may specify where resourcescan be accessed by the APIs, while communicating with the APIsrequesting information from the internal system 206. Similarly, the APPendpoints may be the locations from which applications (e.g.,applications in one or more internal apps 216) may access resources ofthe internal system 206 they need to carry out their functions, and thelogin endpoints may be the touchpoints of any communication pertainingto logins (e.g., authentication and access control) associated with theinternal system 206. For example, the login endpoints may receive andprocess login communications, for internal apps 216 and/or externalconsumer facing applications 208. For some implementations, at leastsome endpoints among the API endpoints, APP endpoints, and loginendpoints, may be tenant branded endpoints, designated to serveparticular tenants 118.

The internal system 206 may additionally include tenant aware servicesand tenant aware transactional data stores. The tenant aware servicesmay include microservices for fulfillment of various CaaS operationshosted in one or more virtual private clouds or one or more physicalprivate networks. For example, the microservices may include, forexample, catalog data, account data, data services, customer servicefunctions, marketing functions, warehouse functions, and/or othersupporting functions. The tenant aware transactional data stores may beone or more databases that store both raw and processed data resultingfrom operations of the tenant aware services, the endpoints, externalsystems 212, and/or tenant data and analytics systems 214. The tenantaware transactional data stores may store, for example, transactionaldata, batch jobs data, searchable data including various analytics andattributes, event messages, and local logs of various raw data.

The internal system 206 may additionally include back office jobs, whichmay comprise instructions, files, or executable applications configuredto perform various back office tasks and/or computing resources toexecute these instructions or applications. The back office jobs may behosted in, for example, one or more virtual private clouds. The backoffice jobs may include, for example, all system components that run andupdate data (e.g., a first order data or any derived data) associatedwith the internal system 206. Such system components may include, forexample, replenishment identifier (RID) generation service(s), sizeadvisor data set, size advisor algorithmic preparation component(s),recommendation service(s), search data sets, etc. The back office jobsmay include, for example, ETL (extract, transform, and load) processesthat comprise collecting data from multiple different sources (e.g.,tenant aware transactional data stores), converting the collected datato other one or more preset formats as deemed necessary, and loading thedata into the target database (e.g., data warehouse systems). The backoffice jobs may also include, for example, periodic data refreshoperations, periodic synchronization states among internal and externalservices, and automated tasks to run in the background at the internalsystem 206.

The internal system 206 may include the file stores for snapshots in oneor more virtual private clouds or one or more physical private networks.The file stores of snapshots may store snapshots capturing states of theinternal system 206 at particular points in time. For example, eachsnapshot may capture settings, files, configurations, and the memorystate of the components of the internal system 206. These snapshots maybe restored upon request or scheduling, and when a snapshot is restored,settings, and/or the state of the internal system 206 may be returned tothe states they were in at the time the snapshots were captured.

The internal system 206 may include third party data ETL processes,which may collect data from different external sources (e.g., externalsystems 212), convert the collected data to other one or more presetformats as deemed necessary, and load the data into one or more targetdatabases (e.g., data warehouse systems and/or tenant data and analyticssystem 214).

The internal system 206 may include the data warehouse systems 230 inone or more virtual private clouds or one or more physical privatenetworks. As discussed above with respect to the back office jobs andthe third party data ETL processes, the data warehouse systems may bethe one or more target databases for ETL processes that collect datafrom various sources (e.g., the external systems 212 or the tenant awaretransactional data stores). The data warehouse systems may then utilizethe collected data as, for example, parameters for business intelligencethat reveals patterns, analytics, and insights for business decisionsassociated with the internal system 206.

The exemplary environment 200 may include tenant data and analyticssystems 214. The tenant data and analytics systems 214 may be computingresources in communication with one or more components of the internalsystem 206 in order to collect, store, and/or manage data and analyticsassociated with the one or more tenants 118. One or more of the tenantdata and analytics systems 214 may be located remotely from the internalsystem 206 (e.g., at tenant servers). The tenant data and analyticssystems 214 may communicate with the components of the internal system206 using API access or file transfer link 254 over one or more networks101. For example, the tenant data and analytics systems 214 maycommunicate with the API endpoints of the internal system 206, orreceive files from third party data ETL processes. The tenant data andanalytics systems 214 may be accessed by employee devices 202, which maycorrespond to employee devices 116 or the tenant devices 120 depicted inFIG. 1.

As shown in FIG. 2, the exemplary environment 200 may also include userdevices 204, which may correspond to the user devices 112 depicted inFIG. 1. Users 108 of the CaaS electronic platform may use the userdevices 204 to access the internal system 206, as recipients of theservices provided by the components of the internal system 206. Forexample, the users 108 of the user devices 204 may be one or moreregistered subscribers who physically receive and wear the items thatare distributed via the CaaS electronic platform. As another example,the users 108 of the user devices 204 may be one or more potentialsubscribers to the CaaS electronic platform. In the context of thecurrent disclosure, a group of such potential subscribers are referredto as a prospect cohort. As shown in FIG. 2, user devices 204 may accessthe internal system 206 via external consumer facing applications 208.The external consumer facing applications 208 may be browser-accessedweb pages or web-based applications that include web-based userinterfaces accessible from one or more user devices 204 over one or morenetworks (e.g., one or more networks 101).

In some implementations, the exemplary environment 200 may include webapplication hosting server 210 to serve as an intermediary for enablingcommunications made between the user devices 204 and the internal system206. The web application hosting server may be an external (e.g. thirdparty) server that provides an online platform for, for example,building a web-based storefront interface and integrating online retailcomponents (e.g., online point-of-sale system) onto the storefrontinterface, for vendors such as the internal system 206. The webapplication hosting server 210 may communicate with the internal system206 (e.g., the API endpoints, the APP endpoints, and/or an employeedevice 202 logged into the internal system 206), to retrieve necessaryinformation about the internal system 206, and to generate ordynamically update an online storefront for the user devices 204. Theuser devices 204 may, in turn, access the online storefront generated ordynamically updated by the web application hosting server 210, throughthe external consumer facing applications 208. In this way, the externalconsumer facing applications 208 may allow user devices 204 to not onlycommunicate with the internal system 206, but also to communicate withexternal systems 212. Such communication with the external systems 212may be enabled by use of one or more API access communication links 250interconnecting the internal system 206 and the external systems 212.

The exemplary environment 200 may additionally include external systems212 (e.g., systems corresponding to the external systems 122 depicted inFIG. 1), that may be accessed by employee devices 202 (e.g., devicescorresponding to employee devices 116 and/or the tenant devices 120depicted in FIG. 1), and also by the internal system 206 (e.g., thesystem corresponding to server system 102 depicted in FIG. 1). Theemployee devices 202 may access the external systems 212 over one ormore networks, using, for example, web browser access or user interfacesincluded in one or more web-based application. Additionally, as shown inFIG. 2, the internal system 206 may also access the external systemsover the one or more networks, using, for example, one or more APIaccess processes 250.

The external systems 212 may include, for example, analytics systems238, A/B testing systems 240, review systems 244, and email marketingsystems 246. The analytics systems 238 may include one or more webanalytics tools provided by an external (e.g., third party) server, thatprovides dashboards, logs, or reports pertaining to, for example,tracking and reporting website traffic for the vendor (e.g., employees116). For example, an analytics tool may be configured to display poorlyfunctioning pages, where visitors came from, how long the visitorsstayed on the website, the visitors' geographical position, visitorsegmentation information, sales activity and performance, and detailedinformation about current visitors. The analytics systems 238 may beaccessed by employee devices 202 (e.g., a vendor of these solutions) to,for example, customize settings, data, and/or configurations, andutilize the analytics data for business intelligence.

NB testing systems 240 may include NB testing tools for measuring andoptimizing user experience by, for example, measuring subjects' responseto variant A against variant B and determining which of the two variantsis more effective. The NB testing tool may be provided by an external(e.g., third party) server. In the context of the CaaS electronicplatform provided by the internal system 206, the NB testing systems 240may perform an A/B test on, for example, a closet interface with avirtual assistant and a closet interface without a virtual assistant.Results of the experimentations may be provided to employee devices 202(e.g., a vendor of these solutions) to, for example, utilize the resultsfor business intelligence.

The review systems 244 may include user review receiving tools providedby an external (e.g., third party) server. For example, a user reviewreceiving tool in the review systems 244 may provide a platform forusers to add reviews, ratings, and/or user generated content such asvideos, to be published for display at the external consumer facingapplications 208. The review systems 244 may be accessed by employeedevices 202 (e.g., a vendor of these tools) to, for example, importreviews for analytics and business intelligence, and/or customizesettings and configurations.

The email marketing systems 246 may include email marketing automationand analytics tools, provided by an external (e.g., third party) server.For example, an email marketing automation tool may maintain mailinglists and mailing schedules, and may modify email marketing messagesbased on what recipients read, click on, or forward. The email marketingsystems 246 may be accessed by employee devices 202 (e.g., a vendor ofthese tools) to, for example, manage communication preferences, thecontent, and/or vendor subscription settings.

The number and arrangement of devices, components, and communicationnetworks shown in FIG. 2 are provided as an example. In practice, theremay be additional devices, components, and/or communication networks,fewer devices, components, and/or communication networks, differentdevices, components, and/or communication networks, or differentlyarranged devices, components, and/or communication networks than thoseshown in FIG. 2. Furthermore, two or more devices shown in FIG. 2 may beimplemented within a single device, or a single device shown in FIG. 2may be implemented as multiple, distributed devices. Additionally, oralternatively, a set of devices (e.g., one or more devices) ofenvironment 200 may perform one or more functions described as beingperformed by another set of devices of environment 200.

FIG. 3 depicts an exemplary method of optimizing user experience, andmore particularly, to optimizing user experience with respect to emailjourneys, according to exemplary embodiments of the present disclosure.In the context of the current disclosure, an email journey may refer toa predetermined set of emails configured to automatically send emailsbased on predetermined triggers. For example, an exemplary email journeymay begin with a welcome email to a user 108 of user device 204. Theuser 108 may be a potential subscriber to the CaaS electronic platform.The predetermined set of emails may be configured to automatically senda follow up email to the user 108 if an action has not been taken by theuser 108 in response to the welcome email after a predetermined periodof time. The predetermined set of emails may be configured toautomatically send a predetermined response email to the user 108 if theuser responds to the welcome email in a predetermined manner. As such,the predetermined period of time and the predetermined manner ofresponse to the welcome email may be examples of the predeterminedtriggers. Accordingly, the exemplary email journey may include thewelcome email, the follow up email, the predetermined response email,and the predetermined triggers, etc. In some embodiments, an emailjourney may be utilized as an automated process of onboarding newsubscribers to the CaaS electronic platform.

In some embodiments, the CaaS electronic platform may utilize threedifferent types of email journeys: (1) a production set email journeywhich may comprise an optimized predetermined set of emails andconfigurations; (2) a quality assurance set email journey which maycomprise the next candidate set of emails and configurations for theproduction set email journey pending validation through further testing;and (3) a development set email journey which may comprise a test set ofemails and configurations. In some embodiments, the email marketingsystems 246 may determine the production set email journey, the qualityassurance set email journey, and/or the development set email journey.It is understood that a combination of one or more components of theexternal systems 212 and/or the internal systems 206 may determine theproduction set email journey, the quality assurance set email journey,and/or the development set email journey in alternative embodiments.

As shown in FIG. 3, a dynamic traffic allocation system 304 may obtaininformation associated with a prospect cohort 302. As described above, aprospect cohort may refer to a group of one or more users 108 of theuser devices 204 who may be potential subscribers to the CaaS electronicplatform. In some embodiments, the dynamic traffic allocation system 304may obtain information associated with a prospect cohort 302 from acertain period in time. For example, the dynamic traffic allocationsystem 304 may obtain information associated with a prospect cohort 302on a daily basis. In such instances, the dynamic traffic allocationsystem 304 may obtain the prospect cohort 302 from the previous day. Insome embodiments, the prospect cohort 302 may be determined based on newemail addresses obtained the previous day via the CaaS electronicplatform. In some embodiments, the information associated with theprospect cohort 302 may include one or more email addresses and/orunique user identifiers (UUIDs). In some embodiments, the emailmarketing systems 246 may include the dynamic traffic allocation system304. It is understood, however, that the dynamic traffic allocationsystem 304 may be a component of the internal systems 206 and/or theexternal systems 121 in alternative embodiments.

In some embodiments, the dynamic traffic allocation system 304 maydivide the prospect cohort 302 into three groups: (1) a development set306, (2) a quality assurance set 308, and (3) a production set 310. Thatis, the dynamic traffic allocation system 304 may split the one or moreusers 108 of the user devices 204 included in the prospect cohort 302into the three aforementioned groups. In some embodiments, the dynamicallocation system 304 may divide the prospect cohort 302 by taking arandom sample of the prospect cohort 302 based on a predeterminedpercentage using a hash function from the UUIDs of the prospect cohort302 and a modulo function. In some embodiments, the UUIDs may becomprised of alpha numeric identifiers. In such embodiments, the dynamicallocation system may convert the UUIDs into numerical identifiers byutilizing the hash function. In the context of the current disclosure,the hash function may be referred to as a mathematical function thatmaps inputs to a value deterministic of the input value. For example, asame output y may be received for any input value x when the hashfunction is performed. In some embodiments, a Fowler-Noll-Vo hashfunction implementation may be utilized as the hash function to performthis mapping. Once the UUIDs have been converted into numeric values,the modulo function may be used to segment the converted UUIDs intomultiple distinct groups. In the context of the current disclosure, amodulo function may be referred to as a mathematical function that findsthe remainder after division of one number, e.g., a numeric version of aUUID, by another number, e.g., a configurable value representing thenumber of desired distinct cohort segments. In some embodiments, theoutput of the modulo function may be applied to a cohort of numericUUIDs, thereby providing a designation, per UUID, of a number between 0and the configurable value divisor. The dynamic allocation system mayallocate prospect groups by enumerating subsets between 0 and theconfigurable divisor value to production, quality assurance, and/ordevelopment sets. While the current disclosure describes dividing theprospect cohort 302 into three groups, it is understood that theprospect cohort 302 may be divided into any number of groups inalternative embodiments. For example, the prospect cohort 302 may bedivided into two groups such as the development set 306 and theproduction set 310.

In some embodiments, the production set 310 may include a larger numberof users from the prospect cohort 302 than the number included in thequality assurance set 308. Similarly, the quality assurance set 308 mayinclude a larger number of users from the prospect cohort 302 than thenumber included in the development set 306. For example, the productionset 310 may include 85 percent of the prospect cohort 302, the qualityassurance set 308 may include 10 percent of the prospect cohort 302, andthe development set 306 may include 5 percent of the prospect cohort302. It is understood that the number of users for each respective set306, 308, 310 may vary in alternative embodiments.

In some embodiments, the production set email journey may be allocatedto the subset of the prospect cohort 302 assigned to the production set310, the quality assurance set email journey may be allocated to thesubset of the prospect cohort 302 assigned to the quality assurance set308, and the development set email journey may be assigned to the subsetof the prospect cohort 302 assigned to the development set 306. In someembodiments, the email marketing systems 246 may allocate the emailjourneys to the respective subsets. It is understood, however, that acombination of one or more components of the internal systems 206 and/orthe external systems 212 may allocate the email journeys in alternativeembodiments. Accordingly, the prospect cohort 302 assigned to theproduction set 310 may experience the production set email journey, theprospect cohort 302 assigned to the quality assurance set 308 mayexperience the quality assurance set email journey, and the prospectcohort 302 assigned to the development set 306 may experience thedevelopment set email journey. The dynamic allocation system may allowprecise control over the risk tolerance for each group, e.g., thedevelopment set 306, the quality assurance set 308, and the productionset 310, due to the ability to configure the divisor used in the modulofunction described herein and the ability to enumerate which subset ofintegers between 0 and the divisor that are assigned to each of thedevelopment set, quality assurance set, and production set. Theutilization of the dynamic allocation system as described herein mayallow simultaneous assurance that each email journey, e.g., theproduction set email journey, the quality assurance set email journey,and the development set email journey, may have an exclusiverepresentation of the prospect cohort 302.

In some embodiments, the production set email journey, the qualityassurance set email journey, and the development set email journey maybe different for each prospect cohort 302, e.g., a first prospect cohortfrom a previous day and a second prospect cohort from a current day. Forexample, the analytics systems 238 may track results in response to afirst production set email journey, a first quality assurance set emailjourney, and a first development set email journey assigned to the firstprospect cohort. Based on the tracked results, the email marketingsystems 246 may determine a second production set email journey, asecond quality assurance set email journey, and a second development setemail journey.

In some embodiments, the tracked results for the first development setemail journey may be compared with the tracked results for the firstproduction set email journey. If it is determined that the trackedresults for the first development set email journey exceed the trackedresults for the first production set email journey, the firstdevelopment set email journey may be determined to be the second qualityassurance set email journey for further verification. If it isdetermined that the tracked results for the first development set emailjourney fall below the tracked results for the first production setemail journey, the first development set email journey may be discarded.In some embodiments, a new set of emails may be determined for thesecond development set email journey.

In some embodiments, tracked results for the first quality assurance setemail journey may be compared with the tracked results for the firstproduction set email journey. If it is determined that the trackedresults for the first quality assurance set email journey exceed thetracked results for the first production set email journey, the firstquality assurance set email journey may be determined to be the secondproduction set email journey. If it is determined that the trackedresults for the first quality assurance set email journey fall below thetracked results for the first production set email journey, the firstquality assurance set email journey may be discarded. Accordingly, thesecond production set email journey, the second quality assurance setemail journey, and the second development set email journey may bedetermined and assigned to a second prospect cohort. In someembodiments, the second prospect cohort may be assigned two emailjourneys. For example, if the tracked result for the first developmentset email journey falls below the tracked result for the firstproduction set email journey and is discarded, there may be no emailjourneys to be assigned for the second quality assurance set emailjourney. In such instances, a second development set email journey and asecond production set email journey may be determined and assigned tothe second prospect cohort.

In some embodiments, the tracked results may include the number of theusers included in the prospect cohort 302 who have signed up. Forexample, the prospect cohort 302 may include potential subscribers.Accordingly, the number of users included in the prospect cohort 302 whohave signed up in response to the production set email journey, thequality assurance set email journey, and/or the development set emailjourney may be tracked. In some embodiments, a threshold may bedetermined for each of the production set email journey, the qualityassurance set email journey, and the development set email journey. Insuch embodiments, the tracked results for each of the production setemail journey, the quality assurance set email journey, and thedevelopment set email journey may be compared with each respectivethreshold in order to evaluate the results. For example, the thresholdfor each of the production set email journey, the quality assurance setemail journey, and the development set email journey may be apredetermined number of signups in response to the respective emailjourneys. In some embodiments, the determined threshold for each of thequality assurance set email journey and the development set emailjourney may be based on the determined threshold for the production setemail journey. While the number of signed up users have been used toexplain the tracked results with respect to the prospect cohort 302,production set email journey, the quality assurance set email journey,and the development set email journey, this is for the purpose ofexplanation only and it is understood that the tracked results mayindicate various measurements in alternative embodiments.

With reference to the development set email journey, the tracked resultsmay be compared with the predetermined threshold for a first developmentset email journey. In some embodiments, the first development set emailjourney may be discarded and a second development set email journey maybe determined if the tracked results fall below the predeterminedthreshold. In some embodiments, the first development set email journeymay be determined to be a second quality assurance set email journey ifthe tracked results exceed the predetermined threshold. In someembodiments, the differences between the first development set emailjourney and the second development set email journey may be determinedin order to isolate one or more factors that may be positively and/ornegatively affecting the first development set email journey and/or thesecond development set email journey. In some embodiments, one or moredevelopment set email journeys may be determined based on the isolatedone or more factors.

With reference to the quality assurance set email journey, the trackedresults may be compared with the predetermined threshold for a firstquality assurance set email journey. In some embodiments, the firstquality assurance set email journey may be discarded and a secondquality assurance set email journey may be determined, e.g., based on aprevious development set email journey, if the tracked results fallbelow the predetermined threshold. In some embodiments, the discardedfirst quality assurance set email journey may be modified. In suchembodiments, the modified first quality assurance set email journey maybe determined to be the second development set email journey. In someembodiments, the first quality assurance set email journey may bedetermined to be a second production set email journey if the trackedresults exceed the predetermined threshold.

With reference to the production set email journey, the tracked resultsmay be compared with the predetermined threshold for a first productionset email journey. In some embodiments, the tracked results of the firstquality assurance email journey may exceed the tracked results of thefirst production set email journey. In such embodiments, the firstproduction set email journey may be discarded and the first qualityassurance email journey may be determined to be a second production setemail journey. In some embodiments, the first production set emailjourney may continue to be utilized as the second production set emailjourney if the tracked results for the first quality assurance emailjourney fall below the tracked results of the first production set emailjourney.

In some embodiments, the prospect cohort 302 may include a number ofusers categorized under one or more different groups (also referred toas “segments”). For example, the one or more different groups mayinclude a no box group, which indicates that the users have not yetreceived a box including wearable items. That is, the no box (No Box)group may include users who have not yet received wearable items via theCaaS electronic platform within a predetermined period of time. Forexample, the No Box group may include users who have not yet receivedwearable items within their first 30 days being a member of the prospectcohort 302. As another example, the one or more different groups mayinclude a no return notified (No RN) group, which indicates that theusers in this prospect cohort 302 have received a box including wearableitems, but have not yet returned the box nor have they provided feedbackregarding the wearable items within a predetermined period of time,e.g., their first 30 days as a member of the prospect cohort 302. As yetanother example, the one or more different groups may include a lessthan 50 percent (LT50) group, which indicates that the users in thisprospect cohort 302 have received a box of wearable items and havefurther indicated that they have used less than 50% of the receivedwearable items within a predetermined period of time, e.g., their first30 days as a member of the prospect cohort 302. As another example, theone or more different groups may include a greater than, or equal to, 50percent (GT50) group, which indicates that the users in this prospectcohort 302 have received a box of wearable items and have furtherindicated that they have used 50% or more of the received wearable itemswithin a predetermined period of time, e.g., their first 30 days as amember of the prospect cohort 302.

FIGS. 4A-4B depict charts utilizing the one or more different groups ofcohorts (i.e., segments) according to some embodiments. As shown in FIG.4A, the chart indicates a conversion percentage for the No Box segment,the No RN segment, the LT50 segment, and the GT 50 segment according tosome embodiments. In the context of the current disclosure, the term“conversion” may refer to users who have converted from a trial periodmember to a registered subscriber to the CaaS electronic platform. Insome embodiments, the trial period may be a period of 30 days. FIG. 4Bdepicts a chart indicating a population breakdown of the CaaS electronicplatform prospect cohort users to the according to the aforementionedsegments according to some embodiments.

FIG. 5 depicts an exemplary method 500 of optimizing user experience,according to exemplary embodiments of the present disclosure. In step502, a first development subset, a first quality assurance subset, and afirst production subset of a first plurality of user identifiers may bedetermined. In some embodiments, each of the first plurality of useridentifiers may be associated with a customer. In some embodiments, thefirst production subset may include a larger number of user identifiersthan the first quality assurance subset, and the first quality assurancesubset may include a larger number of user identifiers than the firstdevelopment subset. In step 504, a first development set of messages maybe determined. In step 506, the first development set of messages may betransmitted to the first development subset of user identifiers. In step508, one or more messages may be received in response to the firstdevelopment set of messages transmitted to the first development subsetof user identifiers. In step 510, a development performance indicatormay be determined based on the one or more messages received in responseto the first development set of messages transmitted to the firstdevelopment subset of user identifiers.

In step 512, a first quality assurance set of messages may bedetermined. In step 514, the first quality assurance set of messages maybe transmitted to the first quality assurance subset of useridentifiers. In step 516, one or more messages may be received inresponse to the first quality assurance set of messages transmitted tothe first quality assurance subset of user identifiers. In step 518, aquality assurance performance indicator may be determined based on theone or more messages received in response to the first quality assuranceset of messages transmitted to the first quality assurance subset ofuser identifiers.

In step 520, a first production set of messages may be determined. Instep 522, the first production set of messages may be transmitted to thefirst production subset of user identifiers. In step 524, one or moremessages may be received in response to the first production set ofmessages transmitted to the first production subset of user identifiers.In step 526, a production performance indicator may be determined basedon the one or more messages received in response to the first productionset of messages transmitted to the first production subset of useridentifiers.

FIG. 6 depicts an exemplary method 600 of optimizing user experience,according to exemplary embodiments of the present disclosure. In someembodiments, method 600 may be an extension of method 500, as indicatedby step A in FIGS. 5 and 6.

In step 602, a second development subset, a second quality assurancesubset, and a second production subset of a second plurality of useridentifiers may be determined. In some embodiments, each of the secondplurality of user identifiers may be associated with a customer. In someembodiments, the second production subset may include a larger number ofuser identifiers than the second quality assurance subset, and thesecond quality assurance subset may include a larger number of useridentifiers than the second development subset. In step 604, a seconddevelopment set of messages may be determined at least based on thedetermined development performance indicator and the determinedproduction performance indicator. In step 606, the second developmentset of messages may be transmitted to the second development subset ofuser identifiers. In step 608, a second quality assurance set ofmessages may be determined at least based on the determined developmentperformance indicator and the determined production performanceindicator. In step 610, the second quality assurance set of messages maybe transmitted to the second quality assurance subset of useridentifiers. In step 612, a second production set of messages may bedetermined at least based on the determined quality assuranceperformance indicator and the determined production performanceindicator. In step 614, the second production set of messages may betransmitted to the second production subset of user identifiers.

In some embodiments, the step of determining the second development setof messages at least based on the determined development performanceindicator and the determined production performance indicator mayinclude: comparing the determined development performance indicator tothe determined production performance indicator; discarding the firstdevelopment set of messages as a result of determining that thedetermined development performance indicator is lower than thedetermined production performance indicator; determining a new set ofmessages; and assigning the new set of messages as the seconddevelopment set of messages.

In some embodiments, the step of determining the second qualityassurance set of messages at least based on the determined developmentperformance indicator and the determined production performanceindicator may include: comparing the determined development performanceindicator to the determined production performance indicator; andreplacing the first quality assurance set of messages with the firstdevelopment set of messages to form the second quality assurance set ofmessages as a result of determining that the determined developmentperformance indicator exceeds determined production performanceindicator.

In some embodiments, determining the second production set of messagesat least based on the determined quality assurance performance indicatorand the determined production performance indicator includes: comparingthe determined quality assurance performance indicator to the determinedproduction performance indicator; and replacing the first production setof messages with the first quality assurance set of messages to form thesecond production set of messages as a result of determining that thedetermined quality assurance performance indicator exceeds thedetermined production performance indicator.

As shown in FIG. 7, a device 700 used for performing the variousembodiments of the present disclosure (e.g., the server system 102, theuser devices 112, the employee devices 116, the tenant devices 120,and/or any other computer system or user terminal for performing thevarious embodiments of the present disclosure) may include a centralprocessing unit (CPU) 720. CPU 720 may be any type of processor deviceincluding, for example, any type of special purpose or a general-purposemicroprocessor device. As will be appreciated by persons skilled in therelevant art, CPU 720 also may be a single processor in amulti-core/multiprocessor system, such system operating alone, or in acluster of computing devices operating in a cluster or server farm. CPU720 may be connected to a data communication infrastructure 710, forexample, a bus, message queue, network, or multi-core message-passingscheme.

A device 700 (e.g., the server system 102, the user devices 112, theemployee devices 116, the tenant devices 120, and/or any other computersystem or user terminal for performing the various embodiments of thepresent disclosure) may also include a main memory 740, for example,random access memory (RAM), and may also include a secondary memory 730.Secondary memory, e.g., a read-only memory (ROM), may be, for example, ahard disk drive or a removable storage drive. Such a removable storagedrive may comprise, for example, a floppy disk drive, a magnetic tapedrive, an optical disk drive, a flash memory, or the like. The removablestorage drive in this example reads from and/or writes to a removablestorage unit in a well-known manner. The removable storage unit maycomprise a floppy disk, magnetic tape, optical disk, etc., which is readby and written to by the removable storage drive. As will be appreciatedby persons skilled in the relevant art, such a removable storage unitgenerally includes a computer usable storage medium having storedtherein computer software and/or data.

In alternative implementations, secondary memory 730 may include othersimilar means for allowing computer programs or other instructions to beloaded into device 700. Examples of such means may include a programcartridge and cartridge interface (such as that found in video gamedevices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to device 700.

A device 700 may also include a communications interface (“COM”) 760.Communications interface 760 allows software and data to be transferredbetween device 700 and external devices. Communications interface 760may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface may be in the form ofsignals, which may be electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 760. Thesesignals may be provided to communications interface 760 via acommunications path of device 700, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems, and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. A device 700 alsomay include input and output ports 750 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

The systems, apparatuses, devices, and methods disclosed herein aredescribed in detail by way of examples and with reference to thefigures. The examples discussed herein are examples only and areprovided to assist in the explanation of the apparatuses, devices,systems, and methods described herein. None of the features orcomponents shown in the drawings or discussed below should be taken asmandatory for any specific implementation of any of these theapparatuses, devices, systems, or methods unless specifically designatedas mandatory. For ease of reading and clarity, certain components,modules, or methods may be described solely in connection with aspecific figure. In this disclosure, any identification of specifictechniques, arrangements, etc. are either related to a specific examplepresented or are merely a general description of such a technique,arrangement, etc. Identifications of specific details or examples arenot intended to be, and should not be, construed as mandatory orlimiting unless specifically designated as such. Any failure tospecifically describe a combination or sub-combination of componentsshould not be understood as an indication that any combination orsub-combination is not possible. It will be appreciated thatmodifications to disclosed and described examples, arrangements,configurations, components, elements, apparatuses, devices, systems,methods, etc. can be made and may be desired for a specific application.Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware. The term “software”is used expansively to include not only executable code, for examplemachine-executable or machine-interpretable instructions, but also datastructures, data stores and computing instructions stored in anysuitable electronic format, including firmware, and embedded software.The terms “information” and “data” are used expansively and includes awide variety of electronic information, including executable code;content such as text, video data, and audio data, among others; andvarious codes or flags. The terms “information,” “data,” and “content”are sometimes used interchangeably when permitted by context.

It is intended that the specification and examples be considered asexemplary only, with a true scope and spirit of the disclosure beingindicated by the following claims.

1-20. (canceled)
 21. A computer-implemented method comprising:dynamically allocating, by one or more processors, a first plurality ofuser identifiers to a first development subset, a first qualityassurance subset, and a first production subset, wherein each of thefirst plurality of user identifiers is associated with a customer;determining, by the one or more processors, a first development set ofemail messages at least based on data regarding an electronictransactions platform; determining, by the one or more processors, adevelopment performance indicator based on one or more messages receivedin response to transmitting the first development set of email messagesto the first development subset of user identifiers; determining, by theone or more processors, a first quality assurance set of email messagesto the first quality assurance subset of user identifiers; determining,by the one or more processors, a first production set of email messagesfor the first production subset of user identifiers; determining, by theone or more processors, a production performance indicator based on oneor more messages received in response to transmitting the firstproduction set of email messages to the first production subset of useridentifiers; and determining a second development set of email messagesby: comparing the development performance indicator to the productionperformance indicator; and as a result of determining that thedevelopment performance indicator is lower than the productionperformance indicator, determining and assigning a new set of emailmessages as the second development set of email messages.
 22. Thecomputer-implemented method of claim 21, wherein the first productionsubset includes a larger number of user identifiers than the firstquality assurance subset, and wherein the first quality assurance subsetincludes a larger number of user identifiers than the first developmentsubset.
 23. The computer-implemented method of claim 21, furthercomprising: determining, by one or more processors, a second developmentsubset, a second quality assurance subset, and a second productionsubset of a second plurality of user identifiers, wherein each of thesecond plurality of user identifiers is associated with a customer;determining, by the one or more processors, a second development set ofemail messages at least based on the determined development performanceindicator and the determined production performance indicator;transmitting, by the one or more processors, the second development setof email messages to the second development subset of user identifiers;determining, by the one or more processors, a second quality assuranceset of email messages at least based on the determined developmentperformance indicator and the determined production performanceindicator; transmitting, by the one or more processors, the secondquality assurance set of email messages to the second quality assurancesubset of user identifiers; determining, by the one or more processors,a second production set of email messages at least based on thedetermined quality assurance performance indicator and the determinedproduction performance indicator; and transmitting, by the one or moreprocessors, the second production set of email messages to the secondproduction subset of user identifiers.
 24. The computer-implementedmethod of claim 23, wherein the second production subset includes alarger number of user identifiers than the second quality assurancesubset, and wherein the second quality assurance subset includes alarger number of user identifiers than the second development subset.25. The method of claim 21, further comprising: transmitting, by the oneor more processors, the first development set of email messages to thefirst development subset of user identifiers; receiving, by the one ormore processors, the one or more messages in response to the firstdevelopment set of email messages transmitted to the first developmentsubset of user identifiers; transmitting, by the one or more processors,the first quality assurance set of email messages to the first qualityassurance subset of user identifiers by assigning a previouslytransmitted development set of email messages as the first qualityassurance set of email messages based on a previously determineddevelopment performance indicator for the previously transmitteddevelopment set of email messages; receiving, by the one or moreprocessors, one or more messages in response to the first qualityassurance set of email messages transmitted to the first qualityassurance subset of user identifiers; determining, by the one or moreprocessors, a quality assurance performance indicator based on the oneor more messages received in response to the first quality assurance setof email messages transmitted to the first quality assurance subset ofuser identifiers; transmitting, by the one or more processors, the firstproduction set of email messages to the first production subset of useridentifiers by assigning a previously transmitted quality assurance setof email messages as the first production set of email messages based ona previously determined quality assurance performance indicator for thepreviously transmitted quality assurance set of email messages; andreceiving, by the one or more processors, one or more messages inresponse to the first production set of email messages transmitted tothe first production subset of user identifiers.
 26. The method of claim23, wherein determining the second quality assurance set of emailmessages at least based on the determined development performanceindicator and the determined production performance indicator comprises:comparing the determined development performance indicator to thedetermined production performance indicator; and as a result ofdetermining that the determined development performance indicatorexceeds the determined production performance indicator, replacing thefirst quality assurance set of email messages with the first developmentset of email messages to form the second quality assurance set of emailmessages.
 27. The method of claim 23, wherein determining the secondproduction set of email messages at least based on the determinedquality assurance performance indicator and the determined productionperformance indicator comprises: comparing the determined qualityassurance performance indicator to the determined production performanceindicator; and as a result of determining that the determined qualityassurance performance indicator exceeds the determined productionperformance indicator, replacing the first production set of emailmessages with the first quality assurance set of email messages to formthe second production set of email messages.
 28. A computer systemcomprising: a data storage device storing processor-readableinstructions; and a processor configured to execute the instructions toperform a method including: dynamically allocating a first plurality ofuser identifiers to a first quality assurance subset, and a firstproduction subset, wherein each of the first plurality of useridentifiers is associated with a customer; determining a firstdevelopment set of email messages at least based on data regarding anelectronic transactions platform; determining a development performanceindicator based on one or more messages received in response totransmitting the first development set of email messages to the firstdevelopment subset of user identifiers; determining a first qualityassurance set of email messages to the first quality assurance subset ofuser identifiers; determining a first production set of email messagesto the first production subset of user identifiers; determining aproduction performance indicator based on one or more messages receivedin response to transmitting the first production set of email messagesto the first production subset of user identifiers; and determining asecond development set of email messages by: comparing the determineddevelopment performance indicator to the determined productionperformance indicator; and as a result of determining that thedetermined development performance indicator is lower than thedetermined production performance indicator, determining and assigning anew set of email messages as the second development set of emailmessages.
 29. The computer system of claim 28, wherein the firstproduction subset includes a larger number of user identifiers than thefirst quality assurance subset, and wherein the first quality assurancesubset includes a larger number of user identifiers than the firstdevelopment subset.
 30. The computer system of claim 28, wherein theprocessor is further configured for: determining a second developmentsubset, a second quality assurance subset, and a second productionsubset of a second plurality of user identifiers, wherein each of thesecond plurality of user identifiers is associated with a customer;determining a second development set of email messages at least based onthe determined development performance indicator and the determinedproduction performance indicator; transmitting the second developmentset of email messages to the second development subset of useridentifiers; determining a second quality assurance set of emailmessages at least based on the determined development performanceindicator and the determined production performance indicator;transmitting the second quality assurance set of email messages to thesecond quality assurance subset of user identifiers; determining asecond production set of email messages at least based on the determinedquality assurance performance indicator and the determined productionperformance indicator; and transmitting the second production set ofemail messages to the second production subset of user identifiers. 31.The computer system of claim 30, wherein the second production subsetincludes a larger number of user identifiers than the second qualityassurance subset, and wherein the second quality assurance subsetincludes a larger number of user identifiers than the second developmentsubset.
 32. The computer system of claim 28, wherein the processor isfurther configured for: transmitting the first development set of emailmessages to the first development subset of user identifiers; receivingthe one or more messages in response to the first development set ofemail messages transmitted to the first development subset of useridentifiers; transmitting the first quality assurance set of emailmessages to the first quality assurance subset of user identifiers byassigning a previously transmitted development set of email messages asthe first quality assurance set of email messages based on a previouslydetermined development performance indicator for the previouslytransmitted development set of email messages; receiving one or moremessages in response to the first quality assurance set of emailmessages transmitted to the first quality assurance subset of useridentifiers; determining a quality assurance performance indicator basedon the one or more messages received in response to the first qualityassurance set of email messages transmitted to the first qualityassurance subset of user identifiers; transmitting the first productionset of email messages to the first production subset of user identifiersby assigning a previously transmitted quality assurance set of emailmessages as the first production set of email messages based on apreviously determined quality assurance performance indicator for thepreviously transmitted quality assurance set of email messages; andreceiving one or more messages in response to the first production setof email messages transmitted to the first production subset of useridentifiers.
 33. The computer system of claim 30, wherein determiningthe second quality assurance set of email messages at least based on thedetermined development performance indicator and the determinedproduction performance indicator comprises: comparing the determineddevelopment performance indicator to the determined productionperformance indicator; and as a result of determining that thedetermined development performance indicator exceeds the determinedproduction performance indicator, replacing the first quality assuranceset of email messages with the first development set of email messagesto form the second quality assurance set of email messages.
 34. Thecomputer system of claim 30, wherein determining the second productionset of email messages at least based on the determined quality assuranceperformance indicator and the determined production performanceindicator comprises: comparing the determined quality assuranceperformance indicator to the determined production performanceindicator; and as a result of determining that the determined qualityassurance performance indicator exceeds the determined productionperformance indicator, replacing the first production set of emailmessages with the first quality assurance set of email messages to formthe second production set of email messages.
 35. A non-transitorycomputer-readable medium containing instructions that, when executed bya processor, cause the processor to perform a method comprising:dynamically allocating a first plurality of user identifiers to a firstquality assurance subset, and a first production subset, wherein each ofthe first plurality of user identifiers is associated with a customer;determining a first development set of email messages at least based ondata regarding an electronic transactions platform; determining adevelopment performance indicator based on one or more messages receivedin response to transmitting the first development set of email messagesto the first development subset of user identifiers; determining a firstquality assurance set of email messages to the first quality assurancesubset of user identifiers; determining a first production set of emailmessages to the first production subset of user identifiers; determininga production performance indicator based on one or more messagesreceived in response to transmitting the first production set of emailmessages to the first production subset of user identifiers; anddetermining a second development set of email messages by: comparing thedetermined development performance indicator to the determinedproduction performance indicator; and as a result of determining thatthe determined development performance indicator is lower than thedetermined production performance indicator, determining and assigning anew set of email messages as the second development set of emailmessages.
 36. The non-transitory computer-readable medium of claim 35,wherein the first production subset includes a larger number of useridentifiers than the first quality assurance subset, and wherein thefirst quality assurance subset includes a larger number of useridentifiers than the first development subset.
 37. The non-transitorycomputer-readable medium of claim 35, the method further comprising:determining a second development subset, a second quality assurancesubset, and a second production subset of a second plurality of useridentifiers, wherein each of the second plurality of user identifiers isassociated with a customer; determining a second development set ofemail messages at least based on the determined development performanceindicator and the determined production performance indicator;transmitting the second development set of email messages to the seconddevelopment subset of user identifiers; determining a second qualityassurance set of email messages at least based on the determineddevelopment performance indicator and the determined productionperformance indicator; transmitting the second quality assurance set ofemail messages to the second quality assurance subset of useridentifiers; determining a second production set of email messages atleast based on the determined quality assurance performance indicatorand the determined production performance indicator; and transmittingthe second production set of email messages to the second productionsubset of user identifiers.
 38. The non-transitory computer-readablemedium of claim 35, the method further comprising: transmitting thefirst development set of email messages to the first development subsetof user identifiers; receiving the one or more messages in response tothe first development set of email messages transmitted to the firstdevelopment subset of user identifiers; transmitting the first qualityassurance set of email messages to the first quality assurance subset ofuser identifiers by assigning a previously transmitted development setof email messages as the first quality assurance set of email messagesbased on a previously determined development performance indicator forthe previously transmitted development set of email messages; receivingone or more messages in response to the first quality assurance set ofemail messages transmitted to the first quality assurance subset of useridentifiers; determining a quality assurance performance indicator basedon the one or more messages received in response to the first qualityassurance set of email messages transmitted to the first qualityassurance subset of user identifiers; transmitting the first productionset of email messages to the first production subset of user identifiersby assigning a previously transmitted quality assurance set of emailmessages as the first production set of email messages based on apreviously determined quality assurance performance indicator for thepreviously transmitted quality assurance set of email messages; andreceiving one or more messages in response to the first production setof email messages transmitted to the first production subset of useridentifiers.
 39. The non-transitory computer-readable medium of claim37, wherein determining the second quality assurance set of emailmessages at least based on the determined development performanceindicator and the determined production performance indicator comprises:comparing the determined development performance indicator to thedetermined production performance indicator; and as a result ofdetermining that the determined development performance indicatorexceeds the determined production performance indicator, replacing thefirst quality assurance set of email messages with the first developmentset of email messages to form the second quality assurance set of emailmessages.
 40. The non-transitory computer-readable medium of claim 37,wherein determining the second production set of email messages at leastbased on the determined quality assurance performance indicator and thedetermined production performance indicator comprises: comparing thedetermined quality assurance performance indicator to the determinedproduction performance indicator; and as a result of determining thatthe determined quality assurance performance indicator exceeds thedetermined production performance indicator, replacing the firstproduction set of email messages with the first quality assurance set ofemail messages to form the second production set of email messages.